Representing Systems with Hidden State

Introduction Learning and planning under uncertainty is a crucial focus of modern AI research. In this paper, we address the issue of sequential decision making. Much of the work in this field is based on the framework of Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998). In this framework, problems are modeled using discrete states and actions. Actions cause stochastic transitions between states. At each time step, a stochastic observation is also generated, based on the current state and the previous action. Much work has been devoted to planning in POMDPs when a model of the system (in terms of the stochastic transitions between states and the probability distributions over observations) is known. Unfortunately, learning POMDPs from data is a very difficult problem. One standard algorithmic solution is expectation maximization (EM) (Chrisman, 1992), but for POMDPs this approach is plagued by local minima (more so than for other probabilistic models) and works poorly in practice unless a good initial model of the system is used (Shatkay & Kaelbling, 1997). History-based methods (McCallum, 1995) often work better in practice but are less general. A lot of recent research has been devoted to finding alternative representations for such systems, e.g., diversity-based representation (Rivest & Schapire, 1994), predictive state representations (PSRs) (Littman et al., 2002) and TD-networks (Sutton & Tanner, 2005). These approaches aim to combine the generality of POMDPs with the ease of learning of history-based methods. The key idea underlying all of these approaches is that the state of the system is not considered as predefined; instead, it is viewed as a sufficient statistic for making (probabilistic) predictions about future trajectories. However, the